A Simple Method for Estimating the Weight Decay Parameter

was initially intended for computing a " quick and dirty " estimate for , which could be used as an initial value for doing a full search using cross validation. The initial estimate, however, turned out to be essentially as good as the final , which rendered further search unneccesary! Another finding that came out of this work, is the fact that guessing the weight decay parameter to be approximately 10 ?2:7 (the average of all values in the experiments) is a surprisingly good estimator. This is very good scaling indeed, given that the problems considered were quite different. This good scaling may explain some of the popularity of weight decay, since it implies that it one does not have to try a large range of values to find a good value for. To give an example of the speed-up achieved by using an early stopping estimate instead of a full search estimate: It takes approximately 45 CPU minutes on a NeXT workstation to produce the histograms in figure 2, and it takes about 100 CPU hours on the same machine to produce the traces in figure 1. However, one would not scan the full log 2 ?6:5; 1:0] region in a real application and this time can be cut by about 80%, since most of the time is spent on overfitting when is very small. The early stopping method is still roughly 25 times faster than the search method.